Method and system to increase operator awareness

ABSTRACT

A method and system to increase operator awareness for process control application providing operators team real time information on controlled process and equipment immersing them into augmented and virtual reality, provided by a computing system which collects and integrates data from process and equipment, leaving equipment, controls, and procedure unmodified. The method is creating an augmented and virtual reality for the operating crew, giving them real time supplementary information based on two types of simulation, one using models and one using process data and learning procedures, together with rich equipment data, equipment locator, enhanced communication and enhanced data acquisition from supplementary sources as surface and airborne robotic and operator headset additional equipment meant to monitor operator bio-parameters and operator&#39;s surrounding environment in order to assure that control is sound and sober, and operators are safe and secure all time providing an optimum high efficiency operation of the controlled process.

STATEMENT REGARDING FEDERALLY SPONSORED R&D

This invention was made with NO Government support.

NAMES OF PARTIES TOA JOINT RESEARCH AGREEMENT

This work was part of research of the mentioned inventors.

CROSS REFERENCE TO RELATED APPLICATIONS

This Application claims no priority.

BACKGROUND OF THE INVENTION 1. Field of the Invention

The present invention relates to a method and device to increase thequality of an industrial plant's control and other complex equipmentused in power production as coal, oil, nuclear, etc., or industry asdone by a team of operators, based on the newest augmented realitytechnologies.

There are various levels of equipment added on the body depending on theneeds and functions one operator has to perform. Virtual reality devicesare allowing operators to see inside the power plant or installation ina case sensitive mode, having various themes, as condition of theequipment, operational equipment, efficiency of use of each device,control system status, inside simulated view of the equipment,temperatures, pressures compliance with the systems, abnormalactivities, predictions for possible future options, etc.

The system consists in a mobile sensor array support, sensors andmicroprocessor system and a plurality of receiving processing stations.

This method consists in a set of procedures to acquire the data, processit and based on evolution trend to predict with anticipation the nextnecessary actions

2. Description of the Prior Art

There are many industrial process control structures, that requireincreased operating crew awareness on the controlled system, from whichnuclear reactor structures, are among the most complex, and with thehighest safety and public opinion impact in case of failure, and that iswhy we paid special attention to nuclear reactors control, but the maingoal is on possible applications to other industrial processes controlsystems, because in spite the accidents in these plants are not asfamous as nuclear reactor mishaps, the damage brought to environment andeconomy are by several orders of magnitude larger, happening with afrequency of few major accidents every year.

An example on complexity of actual process control structures one maysee how GE-Hitachi's amended Advanced Boiling Water Reactor design,downloadable from:https://nuclearstreet.com/nuclear_power_industrynews/b/nuclearpower_news/archive/2011/11/02/nrc-approves-amended-abwr-design-for-new-nuclear-plants-110202It was developed by the now-merged nuclear divisions of General Electricand Hitachi, as well as Toshiba, the 1350-to-1460-megawatt-electric ABWR(Advanced Boiling light Water Reactor) design was first certified by theNRC (Nuclear Regulatory Commission) in 1997 and has been underconsideration for two new reactors at the South Texas Project nearHouston. Although NRG Energy, a key investor in that project, pulled outof the venture, its partner Toshiba continued to seek approval of adesign change to bring the ABWR in line with enhanced safetyrequirements put in place after 9/11.

The actual fleet of world nuclear reactors is 438 nuclear power plantsin operation in the year 2016, and since 1942, when the first nuclearreactor have been commissioned about 4 major accidents two with INES(International Nuclear Event Scale) level of 6 and another two with INESlevel 7 happened, and many minor accidents, and incidents take placeevery year, inflicting material losses to owners and public distress. Infact, in spite death rate is much lower than for road transportationthat scales 30,000 causalities/year in US only, commercial aviation thatis in the range of few hundred per year, every year, due to secrecy,military application and people ignorance, made public more leery onnuclear events than on any other mishap. The only problem is thatcompared to industrial accidents, nuclear reactor accidents are welldocumented, and there is easy to get a lesson learned from there.

The system is indeed very complex, and we paid attentions more toaccidents, as a boost for this patent development because we intended tofind out the triggers of these accidents, the impact of human factor andhow a technological development might help in prevent or minimize theconsequences of such unfortunate events that can be brieflycharacterized that on a previously unknown that is distressed system aperturbation is added that becomes critical and triggers a chain ofcritical events that are countered by underperformance and bad decisionsof the operating crew, which seals the fate of the equipment and ends upin an accident instead of an incident.

Analyzing these accidents we noticed, that at Three Miles Islandfailure, to transmit a lesson learned about reliability of a pressurevalve, that was a main trigger of events, and operators failure tounderstand the correlation between various parameters was what sealedthe fate of the reactor.

It is difficult for someone without at least 10 year experience to knowwhere everything is located. For a new operator, navigating inside thiscomplex labyrinth of rooms full of equipment one might need somethinglike a GPS device with an equipment locater and I would also needsomething to tell what that equipment is and something to tell how touse it and what might happen if I modify the control on it. For exampleit took about 6 hours for the Fukushima team to find a pressure safetyvalve, and open it to prevent a Chernobyl like explosion, but they endup with a hydrogen explosion instead.

In FIG. 1 Nuclear accident brief compared with industrial accidentscausalities as part of the current state of the art technology level ofhazard as part of the state of the art.

On the right ordinate the variation of operational reactors' number isshown, in the dotted curve ending at about 440 in year 2015. It isobserved that this number remained stationary after the Chernobylaccident that acted as a strong deterrent of nuclear power development,many nations seriously reconsidering the nuclear option.

The function was plotted using the thin, continuous line segments, withits values displayed on the right ordinate in logarithmic scale. A trendstraight line is plotted showing that the probability of INES accidentis decreasing as an exponential, in present being smaller than 1 in 1000reactors per year. The probability of high severity accidents basicallyfollows the general trend, making the actual wait time to the next majoraccident longer than 40 years, probably by 2040s, if nothing outstandinghappens.

In spite industrial accidents' causalities are by more than an order ofmagnitude higher than those coming from nuclear power, public's exigencyis increased towards nuclear power, due to its hard to understandmanifestations and its scary debut, associated to past and presentmilitary use.

Human factor is determinant in sealing the fate of an installationundergoing a series of critical events, and that is why at the heart ofthe DOE's Research and Education Facility is the Nuclear Power PlantMain Control Room (NPP MCR) simulator room, where some of the keyfeatures include:

a) Reconfigurable Modular Consoles

b) Simulator System Hardware and Interface Setup

c) The Human Factors Observation and Monitoring Gallery

d) Human Performance Measurement Systems

e) Integrated Instrumentation and Control Systems

On the web pagehttp://www.kepco-enc.com/korea/sub.asp?Mcode=F01000&BID=B44&idx=105&BoardType=view,where Tecnatom is participating in the supply of the control rooms forreactors 3 and 4 of the Fuqing nuclear power plant, in China, and hassigned a new contract to participate in the supply of theinstrumentation, process systems and control rooms of the two new groupsof the Chinese Fuqing nuclear power plant.

That scope of the works includes the following:

Engineering of the Man-Machine Interface and, as a result, design of theoperating screens and of the digital Control Room itself and of theremote shutdown panel, and has improved design and development of theOperating Aid Systems: the computerized Operating Procedures andAdvanced Alarms system.

The awarding of this contract, which comes in the wake of those forFuqing 1 and 2, Fangianshan 1 and 2 and Hainang 1 and 2, means a new andimportant backing for Tecnatom and makes the company a reference atinternational level in the field of new reactors.

This design seems more advanced than what we found in the other powerproduction plants futuristic designs, or general industrial automationwhere evolutionary little changes are preferred to game changingsolutions, but fails to solve the main difficulty for the operator,consisting in watching in real time the evolution of several parametersin order to control the process.

The most difficult aspect our invention wants to solve is correlationand interpretation of the indications of the instruments placed in suchpanels is difficult for many and very complex for those who know. As yousee, the operators in both cases, no matter if panel is older or newer,is pointing hands in different direction and looking in another, even ina most advanced virtual room. See:http://phys.org/news/2013-08-virtual-room-nuclear-industry.html

Virtual control room helps nuclear operators, industry is where TheDepartment of Energy's new Human System Simulation Laboratory at IdahoNational Laboratory is a full-scale virtual nuclear control room thatcan test the safety and reliability of proposed technology replacementsbefore they are implemented in commercial nuclear control rooms, may beseen on: https://www.pinterest.com/insensertech/control-rooms/These arethe modern designs of the future control rooms, but the problem remainsthe same as shown in the right side interrelation between severalindication spread on the screens.

Examples of stereoscopic images may be used to create virtual realitygiving a very good sensation of 3D visualization. In the upper leftcorner, there is an example of Google Cardboard. The image is split into2 images https://developers.google.com/cardboard/Cardboard aims atdeveloping accessible virtual reality (VR) tools to allow everyone toenjoy VR in a simple, fun, and natural way. The Cardboard SDKs forAndroid and Unity enable you to quickly start creating VR apps or adaptyour existing application for VR.http://www.theverge.com/2014/6/25/5842188/googles-cardboard-turns-your-android-device-into-a-vr-headset

The result is Google Cardboard, which in its current state requires someand cannot simply be bought as a completed product. In order to build ityou need a pair of lenses with a 40 mm focal distance to keep thephone's screen in focus. The kit also requires magnets, Velcro, a rubberband, and an NFC tag if you want to tap your device to the headset tolaunch the app right away. The magnet and rubber band serve as amakeshift hardware button for your phone, something decidedly analog.https://forums.oculus.com/viewtopic.php?t=5800&start=240

Representation of how oculus prototype “Crystal Cove” is made may beseen athttp://www.hypergridbusiness.com/2015107/product-review-sunnypeak-virtual-reality-headset/Samefunction as Google Cardboard except made of plastic and with morefunctions.https://www.osapublishing.org/aop/fulltext.cfm?uri=aop-5-4-456&id=274728Color-interlaced anaglyph stereo, where color-code 3D is a newer,patented stereo viewing system deployed in the 2000s by Sorensen et althat uses amber and blue filters that can be seen at:http://www.augmentedrealitytrends.com/augmented-reality/3-amazing-ar-vr-innovations-mwc.html

The concept is still in its formative years and it depends on thermalimaging cameras to track the residual heat that is left by our body onany surface that it touches. The heat signatures from our fingerprintsare used by Metaio as a means to input data.

Nokia reveals new City Lens augmented reality app for Windows Phone 8lineup handset. http://www.engadget.com/2012/09/11/nokia-reveals-n

New to the Lumia 920 and Lumia 820, Nokia has announced a refinedversion of City Lens replete with 3D icons and the ability to disablesuggestions that aren't within the line of sighthttp://www.craveonline.com/design/811705-5-reasons-google-glass-failed-miserably#/slide/1Virtual and Augmented reality applications started to appear, butapplying it in industrial environment is a more complex responsibletask, because it has to be reliable easy to use, and fail safe.

DEFINITIONS

Virtual reality—the computer-generated simulation of a three-dimensionalimage or environment that can be interacted with in a seemingly real orphysical way by a person using special electronic equipment, such as ahelmet with a screen inside or gloves fitted with sensors.

Augmented reality—a technology that superimposes a computer-generatedimage on a user's view of the real world, thus providing a compositeview.

-   -   Stereoscopy—sometimes called stereoscopic imaging, is a        technique used to enable a three-dimensional effect, adding an        illusion of depth to a flat image. Stereopsis, commonly (if        imprecisely) known as depth perception, is the visual perception        of differential distances among objects in one's line of sight.

CASL—is The Consortium for Advanced Simulation of Light Water Reactors(CASL) was established to provide leading edge modeling and simulation(M&S) capability to improve the performance of currently operating lightwater reactors.

LIDAR—is a surveying technology that measures distance by illuminating atarget with a laser light. Although thought by some to be an acronym ofLight Detection And Ranging, the term LIDAR was actually created as aportmanteau of“light” and “radar”.

Holistic—relating to or concerned with complete systems rather than withindividual parts

bio-parameter is referring to biometric and medical applications wherebody monitoring is needed by using specialized sensors as: pulse, oxygenin blood (SPO₂), airflow (breathing), body temperature,electrocardiogram (ECG), electromyogram (EMG), electro-encephalogram(EEG), skin conductivity (sweat), glucometer, galvanic skin response(GSR—sweating), blood pressure (sphygniomauucter) mid operator body andbody-parts position (accelerometers), and other measurements as: sound,vibration, electric and magnetic fields, temperature, light, in amodular structure used depending on local needs and requirements.

environment parameters is referred to external to operator as airtemperature, heat index, barometric pressure, relative humidity,radiation on various spectral intervals as radio, microwave, infrared,visible, ultraviolet, x and gamma, as well as electric and magneticfields (compass), sound and vibration and their directions

Headset—refers to virtual or augmented reality headset or helmet, whichis a specialized, head-mounted display device that provides a simulatedvisual environment through physical display optic lenses, allowing theuser to see both a digital display and/or the world through the glasses,which will be upgraded with supplementary equipment to meet ourrequirements.

COMSOL Multiphysics® is a general-purpose software platform, based onadvanced numerical methods, for modeling and simulating physics-basedproblems.

Finite element analysis (FEA) is a computerized method for predictinghow a product reacts to real-world forces, vibration, heat, fluid flow,and other physical effects. Finite element analysis shows whether aproduct will break, wear out, or work the way it was designed.

fluid dynamics is a sub-discipline of fluid mechanics that deals withfluid flow—the natural science of fluids (liquids and gases) in motion.It has several sub-disciplines itself, including aerodynamics (the studyof air and other gases in motion) and hydrodynamics (the study ofliquids in motion)

Robotics is the branch of mechanical engineering, electrical engineeringand computer science that deals with the design, construction,operation, and application of robots, as well as computer systems fortheir control, sensory feedback, and information processing

CAD—Computer-aided design (CAD) is the use of computer programs tocreate two- or three-dimensional (2D or 3D) graphical representations ofphysical objects. CAD software may be specialized for specificapplications.

real-time—is understood as the actual time in which a physical processunder computer study or control occurs, or a time required for acomputer to solve a problem, measured from the time data are fed in tothe time a solution is received, where timing or arrangement is allowinga process, to occur normally, as without delay or asynchronous, insimple words, is a relative time interval that allows a control systemenough time to control a process efficiently.

SUMMARY OF THE INVENTION

The present invention is about a method to increase awareness of theoperators that control an industrial process by providing them a highquality immersive augmented reality system that collects data from theinstrumentation they have, and transforms it into images with color,pulsation, and applies over the image of the desired equipment generatedby a computer acquired design after equipment's blueprints, or over theimage transmitted from a fixed or mobile camera. When camera is mobile,carried by a flying or surface robotic system, or human operator, it mayhave supplementary measurement capabilities. The system is able to addthem on the measured part, complementary to the instrumentation alreadyin place. A data acquisition system is connected with the existentcontrol room equipment in a non-invasive manner, and collects the datathe operator sees on various panels and displays, and transmits it to asuper-computer that is doing, real-time data integration. This systemdoes not change the control room, but adds to it, improving operator'sreal time awareness on process they control, helping them make optimumdecisions.

This system also collects and augments the communication system,integrating images from all surveillance and technologic cameras, aswell from all sensors in the surroundings of the reactor, prioritizingit.

A 3D schematic of the equipment is generated using the building andequipment design files. The indications are placed on each device andthe corresponding data from near-by instrumentation is added in afriendly graphic representation,

The operator may visualize the process on a flat screen stereoscopicimager or in a pair of stereoscopic display goggles, having thecapability to host several operators in parallel looking at variousdevices and modules with capability of zooming-in and generatingdetailed explanations. Supplementary, simulations from codes developed,will be added, if allowed, using the instrumentation data, to generatethe particular simulations and to benchmark it.

The main images generated by the system will be related to the normalityof operation, inside safety limits and designed performance parametersthat may also be acquired from history/data log of the previousoperations. The computer will compare the actual parameters correlationwith those from previous operations, and highlights the discrepancies.

At user's, or operator's request it can produce a 3D global view of thewhole assembly, in color code signaling with blinking colors the areaswith irregularities, as having mal-operation or different thanpreviously observed operation. Simulation codes develop predictivemaneuvers, calculating the effects of any possible change, using it asadvice for the operator. It uses a simulation code based on adaptivelearning, by monitoring previous operation parameters, forming databasesand deducing potential inter-correlations, which are further used tomake guesses and compare with what has been obtained improving thepredictions, as it goes.

It may generate several types of images as the on/off equipment map,localize an equipment by request, for example: “Where is the pressurerelease safety valve?”, a piece of important information in Fukushima'scase, where it took a flock of operators several hours to find andactuate it, with minimal success.

It may generate charts showing the evolution of various parameters, andmay generate images showing where and how to localize themal-functioning equipment.

This equipment may be added to the actual power plants, with a minimuminvestment, helping to navigate the reactor towards high efficiency andhigher safety, integrating in customized applications all theaccumulated knowledge, and using as platform for further simulationdevelopment. Other control units from chemical and reprocessing industrymay also benefit using this equipment. Communication inside the controlroom and with external workers was among first to fail in the case ofThree Miles Island and Chernobyl, therefore the present system providesassisted communication and direct access online to operation data, inconditions of internet secure applications.

The system is conceived having disaster in mind, and it is designed tobe robust and monitor itself for integrity using redundancy as anembedded feature.

In this concept, human factor is important and human decisions may sealthe fate of an equipment and sometimes of the entire surrounding area,and that is why monitoring that operating crew is safe and sound is animportant function. The head sets are not simple augmented realitydevices, as those on the market now, but they acquire and transmit inreal time the biometric parameters of operators, in a modular approach,starting from vitals, to more elaborated functions as electro-graphs ofvarious functions of the body as hart, brain, muscles.

It also records in each location the surrounding environment data, fromall-around sound and image, to temperature, weather data and other addedparameters in case sensitive mode as radiation, gas composition asleaked gases for toxicity or explosive atmospheres, and various objectrecognition in case of drastic changes in the plant environment as afterdisasters or terrorist acts, features supposed to allow operators toproceed safely with the process to the nearest safe regime.

The system is a modular open system, and any supplementary requiredfunction may be added, or its complexity may be reduced to a minimumnecessary.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a view of the nuclear accidents compared with commercialaviation causalities;

FIG. 2 shows an augmented and virtual reality, 3D immersivevisualization plant's control device;

FIG. 3 shows the schematics of networking and communication system fordistributed Augmented Reality with fixed and mobile Virtual realitydevices and infrastructure

FIG. 4 details how an augmented reality mobile device on operator onmove recalibrates using a control panel at an entry port when operatorgoes inside controlled infrastructure with restricted communication

FIG. 5 describes a mobile operator bio-parameter monitoring system thatis distributed on operator's body,

FIG. 6 gives details on mobile operator environment monitoring system

FIG. 7 describes an operational situation, with 3 rooms, where thewireless communication has limited propagation

FIG. 8 shows a computer system controlling the information flow witheach headset and operator system using plant and cloud computers.

FIGURES DETAILS

FIG. 1—is a view of the major accidents compared with nuclear accidentsand commercial aviation causalities;

-   100—Chart representing the growth in industrial processes and    accidents as function of time;-   101—Timeline, Year;-   102—Ordinate with values for various functions in linear scale—left    scale;-   103—Ordinate in logarithmic scale-right scale;-   105—The evolution of number of various technologic processes that    needs to be controlled in [thousands units] read on right scale;-   106-Number of death per year in various industrial major accidents    [thousands] on the left scale;-   107—Number of nuclear power reactors [units] on right scale;-   108—Nuclear accident INES level on the left scale;-   109—Calculated probability of accident function normalized at 1000    units, read on the right scale;-   110—Trend function line for the probability of nuclear accident read    on the right scale.

FIG. 2—Augmented reality 3D immersive visualization system block diagramand control device.

-   200—Augmented reality headset;-   201—Display;-   202—Left eye;-   203—Right Eye;-   204—Lenses;-   205—Left image;-   206—Right image;-   207—Localization, attitude device (gyro-accelerometer-compass    circuit);-   208—Left imager (camera, LIDAR, sonar);-   209—Right imager (camera, LIDAR, sonar);-   210—Object of interest;-   211—Augmented reality scenes—details;-   212—Menu option bar for augmented reality;-   213—Pointing or other operator communication with compute device;-   214—Cloud —computer Wi-Fi communication;-   215—Operator's position in front of control panel or instrument;-   216—Operator bio-parameter monitoring sub-system;-   217—Operator's environment parameters monitoring sub-system;-   218—Control room;-   219—Data bus;-   220—Virtual and augmented reality plant's visualization and control    sub-system;-   221—Overall view of the plant operation;-   222—Detailed view of various controlled processes;-   223—Holistic view of plant connected to grid and environment;-   224—Computer system integrating data, simulation, and specialized    visualization requests;-   225—Simulation of various processes and equipment operation in plant    to augment current measurements;-   226—Memory of operating parameters in the previous regimes;-   227—Simulating system using previous operating regime data;-   228—Instrument positions log, in plant's coordinates;-   229—Building and installation plans;-   230—Integrated, redundant communication sub-system;-   231—Image, sound, vibration monitoring sub-system;-   232—Data acquisition and processing sub-system;

FIG. 3 shows the schematics of networking and communication system fordistributed Augmented Reality with fixed and mobile Virtual realitydevices and infrastructure

-   300—virtual reality visualization enclosure;-   301—Computer producing VR images, triggered by operator's request    for analysis;-   302—operator team members;-   303—VR terminal, specialized computer;-   304—process monitoring system, computer system;-   305—current visualization session;-   306—emergency interruption generated by the process computer;-   310—augmented reality mobile devices;-   311 internal location coordinates system, specialized computer;-   312—operators' location;-   313—operator's headset line of sight;-   314—communication with computer devices;-   315—computer database that is storing and processing previous    operation history;-   316—process and instrument database specialized computer;-   317—operator menu selection mode and simulation computing system;-   318—example of instrument generating process data object of    interest;-   319—redundant communication system;-   320—operator team members;-   321—operator team members;-   322—operator team members;-   323—operator biometric parameters and surrounding environmental    parameters measuring sub-system and hazardous conditions life    support;

FIG. 4 details how an augmented reality mobile device on operator onmove recalibrates using a control panel at an entry port when operatorgoes inside controlled infrastructure with restricted communication

-   400—operator;-   401—operator's arm-   402—access area, entry location/calibration system-   403—room/area's coordinate system fixed reference operator's pad-   404—cable/wire connection-   405—wi-fi/radio connection-   406—IR connection-   407—calibration mode button-   408—area's coordinate sistem's fixed reference-   409—operator's location vector in area's coordinate system-   410—operators head/helmet coordinate system-   411—augmented reality mobile device, headset (310 in FIG. 3)-   412—RFID-   413—GPS-   414—3 axis gyroscope-   415—3 direction inclinometers-   416—3 directions accelerometers-   417—3 direction magnetometers-   418—EM spectrum analyzer/receiver-   419—ultrasound distance measurement/ultrasound locator (proximity    sonar)-   420—Radar-   421—Lidar and as a telemetry detector as radar, laser detector-   422—stereoscopic camera in various bands as visible, filtered,    IR-night vision or IR-thermo-vision, placed on operator

FIG. 5 describes a mobile operator bio-parameter monitoring system thatis distributed on operator's body

-   500—mobile (with AR headset) operator;-   501—fixed reference area pad;-   502—in front of the entry wall;-   503—operator's arm;-   504—reference coordinates calibration button;-   505—room reference system of coordinates;-   506—location vector relative to area coordinate system;-   507—augmented reality headset;-   508—EEG measurement;-   509—a skin conductivity measurement;-   510—Oxygen level measurement;-   511 breathing gas/life support;-   512—temperature measurement devices;-   513—measuring Breathing rate;-   514—a pulse rate/EKG;-   515—position of body components measurement;-   516—EMG measurement;-   517—pH measurement.

FIG. 6 gives details on mobile operator environment monitoring system600 mobile operator;

-   601—Operator's hand;-   602—pointing system and communication interface at operator choice,    as joystick, eye tracker, as part of augmented reality gear,    including voice recognition.-   603—Operator headset-   604—external temperature field sensors;-   605—multi-gas monitoring sensor;-   606—humidity sensor,-   607—sound surround monitoring sensors;-   608—radiation type and level measurement;-   609—EM spectrum monitoring;-   610—Exo-skeleton control parameters;-   611—vibration monitoring sensors;-   612—supplementary life support gear parameters monitoring for air or    oxygen pressure, temperature, humidity;-   614—hazmat suit including power supply control parameters for    assisted heating or actuators;

FIG. 7 describes an operational situation, with 3 rooms, where thewireless communication has limited propagation

-   700—first room;-   701—second, adjacent room-   702—first room's coordinate system,-   703—entry location system-   704—operator's hand touching calibration button-   705—operator in first room-   706—door of first room-   707—direct communication cable-   708—shielding/obstructing wall that negatively affects communication    from near by first room-   709—third room with no coverage from the first two rooms-   710—coordinate calibration entry point in other room-   711—network connection wire-   712—coordinate reference system in third room-   713—operator's position vector in another room-   714—operator's hand touching calibration button-   715—operator in another room-   716—direct communication cable

FIG. 8 shows a computer system controlling the information flow witheach headset and operator system using plant and cloud computers.

-   800—computer system controlling the information flow to operators-   801—headset with augmented reality display-   802—operator system-   803—multiple use plant's computers-   804—multiple use plant's computers-   805—multiple use plant's computers-   806—communication system of computers-   807—data acquisition sub-system-   808—computer cluster holding data base

DETAILED DESCRIPTION OF THE INVENTION

The inventors consider the developments in wearable electronics and of3D virtual and augmented reality devices as well the successes inmodeling the industrial processes down to minor details. It also takesinto consideration the fact that most of the actual developments invirtual reality and augmented reality require a large computing power,and may not be useful for operators that have to make split-seconddecisions, being more useful for finding the optimum-optimorum of asteady process than of a fast transitory one. The experience ofCASL-VERA developments, in nuclear power plant control, showed that inspite these visualizations are high-quality, high detail, it requires asuper-computer to obtain them, may not be available in real time, orclear enough when needed, and a database with simple equipment behaviorextrapolations is desired as a first resource and for redundancyreasons.

It will come as a requirement or recommendation, to develop redundantcommunication systems as well as redundant power supply systems in orderto be able to maintain a holistic knowledge and process controlfunctions as long as possible during high severity events, or disasters,making the control process bring no contribution or to increase theimpact factor, by aggravating the consequences/outcome of an accident ormalfunction.

Having in mind that this system have to be operational in crisis time,that the ambient where augmented reality devices work, might not bealways nice and cozy the headset devices have operator and operatorenvironment monitoring features as well as locating capabilities,specifically designed for industrial environment where satellite or cellphone, GPS might not work, radio communication may be affected by thestructure of building and equipment in the same way a Bluetooth or Wi Fiis affected inside homes, each plant will have its own, redundantcommunication system, from, wire, optical to wireless.

2. Best Mode of the Invention

FIG. 2 shows the devices in of the best mode contemplated by theinventors of the use of a data acquisition and processing system. Itsaccessories devices according to the requirements are presented in FIG.2 with some solutions and developments that are embedded in the presentinvention.

The invention corrects previous deficiencies of the complex processmonitoring and control devices, which are bulky, hard to correlate,distributed in terrain and in control room, improving their performanceswith respect to the usefulness of the data, as follows;

-   a)—Improves the usefulness and comfort of the bearing an augmented    reality equipment, connected via WiFi to a system of distributed    emitter receiver hubs, that assures a strong audio and visual    communication among the crew members from peer to peer and    hierarchically, that may be baked-up by IR and wire communication;-   b)—Makes a system that warns operator when something runs out of    range and is about to take an action, as a result of an undesired    evolution of the controlled process, aided by simulating    capabilities, and embedded “expert functions” in the computer    system;-   c)—Is easy, upgradeable being modular in structure, and having a    virtual reality control room (one or several) upon the needs, and    several augmented reality devices as wearables on operators;-   d)—Has a ultra light sensor applicator on operator's body structure,    easy removable, and with self-control of good operation,    transmitting also operator's bio-parameters related to operator's    state of wellbeing;-   e)—Is redundant, in order to provide needed data, in the most    difficult circumstances, as an accident, nuclear or classical,    triggered by adverse nature actions or by flaws embedded in    technology, operating in distressed areas.-   f)—Is developed in various functional approaches, from the threshold    detection to anticipation, with different complexities and    redundancy level, in agreement with the necessity, being more    complex for a nuclear power plant than for a brewery;-   g) Improves the warning and alert to the operator, by early    detecting any anomalous evolution, based on customized data sets of    the same system previously operating.

Best application of the invention is explained in FIG. 2, but it is notlimited to specific application presented and there are also someapplications that do not require such complex equipment, and asimplified version is possible to be used, and gradually upgraded. Itmay start with versions that does not anticipates the critical momentsbased on advanced simulation on supercomputers, that may bring themaximum efficiency to a operator minimizing his run outside the normalparameters.

3. How to Make the Invention

As one can see from the drawings this method and procedure includes adevice that is conceived to prevent the most complex accidents actuallyknown in industrial plants, being usable in any other process controlunit, being a powerful information unit, helping the operation crew keepthe plant safe and on its optimum.

The application of this system does not require any modification to theactual controls and procedure, and it is simply added over the existentprocess control system.

It is made of an optimized data acquisition that transmits the data to acomputer that calculates the evolution, with anticipation, predictingthe best regimes, or in the case when something bad seems to happen, itwarns operators, making them aware on the issue and potential option tododge a bad outcome.

The virtual reality enclosure, 220, is built with the purpose of theproject is to have at least one person look inside all of the structureas in a doll house similar to a supervisor, using various visualizationfunctions. That to have all of the operating details necessary forexplanation for any instrument or module component of a control box andresults of complex simulations that may predict the status and futureevolution of the controlled process (reactor power plant) on mobileaugmented reality devices.

Therefore all operators will be connected, and act all like one, inperfect coordination.

A high capability computing system, 224, is in charge with presentationof the entire information for operators, supervisors and approvedexternal, remote links, where customized augmented reality istransmitted to user's visualization device, generally that is preparedby a super-computer that integrates a large variety of data withsimulation.

This computing system will generate an overall view, 221, of the entirepower plant in 3D semi-transparent view, or 2D images, being casesensitive, making the supervisor and operators aware of the way thesystem is operating and may highlight issues where something is notoperating or working right or an abnormal behavior is obtained withrespect to previous good and safe operations.

A high importance and attention have to be given to the process core,222, where the operating is complicated by the fact that one has tounderstand processes inside the core and foresee the effect of touchingany control button.

A great support, in the case of a nuclear power plant may be provided bythe progress made in the nuclear reactor simulation research and resultsof CASL (Consortium for Advanced Simulation of Light Water Reactorsfounded by DOE in 2010), 225, initiative that may allow a supercomputerto predict the effect of any change in control system, or temperature,and composition of materials inside the reactor core. Similarsimulations are available for classical power plants, boiler, furnaces,chemical reactors, etc. that may be integrated in order to enhance theinformation supplied to human or automatic operators.

Any controlled process of some importance to us (say a nuclear orcogeneration power plant) is not an insulated system, 206, that worksalone, but it is connected to electricity grid where it delivers itselectric production and/or takes power for its operation and is alsoconnected to water sources and is influenced by weather and other events(social unrest, security, equipment malfunction etc.). That is why aholistic real-time knowledge for operators, have to consider all thesefactors and it is good to have everything accumulated and integrated.

A database containing records of parameter evolution during previousoperation, 226, is very useful information rendering function of allinstruments and modules where a prediction for a future regime may beobtained from memory of operating parameters in the previous regimes.The device will take previous history of parameters and compare with thecurrent parameters, if something is different, it will make the operatoraware of a possible issue/or problem.

A distinct, redundant module, 227, takes previous history records fromdatabase, 226, and compiles them into possible events in the future.This system may work autonomously from computer that is providing datato supervisor. A simple case may be that related to reliability issue,pre-failure symptoms and lessons learned, that are taken from thedatabase and translated into future events with pre-warning signals andexplanations.

The augmented and virtual reality systems, 228, is using design localplant's plans, blueprints, 229, and compiles them into a type of 3D mapin order to accurately know where everything is located, how it lookslike, what is it function and more, in order to be used to augment thereality of the mobile operators looking for that object, or to create avirtual reality. Here we will not describe all the functions thishybridization may provide, but is desired that any information anoperator on spot needs to know, to be available at that instance. Anexample of information may be provided is Instrument opticalrecognition, instrument bar code identification or RFID reading, whatinstrument is doing, where is connected, its cabling to the transducer,its electronic and mechanical design, date of last revision, whatcomputer predicts the reading should be compared to its actual reading,limits of operation, record of possible incidents with similarinstruments, repairing or control procedures, etc.

Communication and synchronization among team members is a criticalissue, especially in distressed systems. A well-defined communicationstructure, 230, that allows real-time communication among team membersdistributed across the building to provide them with all necessaryinformation or data for their augmented reality, and operation is vitalfor the well-being of a controlled process or installation. In thenuclear accidents history that was sometimes the trigger of anescalation of the accident status from bad to worse. It is known thatinside industrial buildings communication does not work as in plainfield, and special care has to be taken to deploy the rightcommunication system. The handset will be provided with optic andelectric cables to connect in various ports, in case of jamming, and WiFi and IR wireless communication. Headset location system will usebuilding coordinates, and local references and calibration instead ofgenerally used coordinates as OPS. Hierarchically structuredcommunication, with liaison points, or peer to peer, or allsimultaneously will be implemented as a function of circumstances.

Image, sound and vibration recording and interpretation system, 231, isimportant. Receiving images would allow the operator to see presence invarious areas and temperature distributions on various modules usinginfrared thermo-vision cameras. Night-vision on headsets will be animportant feature, making the crew able to safely operate duringblackouts. Sound and vibration would allow the operator to hear if apump is malfunctioning or to detect any abnormal noise in the system,and localize its origin. Headsets will transmit their sound and imageinformation to computer analyzer that will extract the background noiseand search for abnormal noise, or background modifications, and makeoperators aware.

Data acquisition, 232, is not a trivial task inside a power plant orother process, in spite modern equipment have these indications digitaland present in various control units, bringing all data in real timeinto a single machine still remains a challenge. No matter thegeneration of the control room, a supplementary data acquisition systemwould collect all the values of the instruments and submit to thesupercomputer unit in order to use them to create augmented reality. Theinstallation may also be used to benchmark various simulation codes,because these codes may have available same inputs and their predictionshave to be compatible with measurements, or the differences will bevaluable indicators on what is missing or malfunctioning inside a code.

Control room, 213, no matter the generation of construction and type ofinstrumentation, will be preserved. This system and associated devicesare not intended to replace control room, replace or augment itsfunctions, or change the operating procedures. Procedures would remainas usual except supplementary level of knowledge and coordination willbe introduced that may guide the operators with the augmented realityattached on their displays or vision devices (a type of Google Glass)Operator positions, 215, will remain same, will have to do sameprocedure will only difference is that operators are now more aware ofthe status of the plant and the effect of adjustments, wearing augmentedreality device when needed.

in its simplest version presented in FIG. 2, the system is at itsminimal level, when advanced simulation software unit is missing,predictions for the future evolution being made only based on databaseand lessons learned.

In order to make the invention, one have to add the virtual andaugmented reality system, as presented in FIG. 2 and customize it forthe specific process intended to be controlled, introducing inside database all the necessary details, enhance the instrumentation,communication and localizing system at the level of hazard anticipated,test it and provide necessary training to operators.

DETAILED DESCRIPTION OF THE FIGURES

The technologic processes control systems have a very large diversity,and an augmented reality control system have to be customized for eachprocess, such as even an unexperienced operator to be able to safelycontrol the process and drive the equipment to a safe rest, and face themost difficult challenges that usually cripples the equipment ending inan accident. But, the major gains of application of this device arecoming from normal operation, where it helps operators maintain theprocess on its optimal range all the time, detect in advance potentialfailure and schedule optimal maintenance.

FIG. 1—is showing a plot of the accident data in industry and nuclearpower, from where we learned something and this idea was triggered,giving a brief view of the major industrial accidents compared withnuclear accidents, where we have compared the number of fatalities fromgeneral industry accidents with the INES (International Nuclear EventSeverity scale) level associated to nuclear power plants accidents,using a chart, 100, representing the growth in industrial processes andaccidents as function of time. In fact, nuclear power plants are veryfew, 439 operating now on the entire world, they are not really our maincustomer, but they have very well documented accidents, compared withthe industry where the fuzziness and cover-ups makes it hard tounderstand the magnitude, real causes and triggers.

On abscises we represented time line, 101, of the events in years, andwe have used two ordinates, one linear, 102, ordinate, on left side,with values for various functions including INES (International NuclearEvent Severity Scale), industrial accidents fatalities are representedin linear scale on the left side.

On the right side we plotted values on a multiscale, 103, an ordinate inlogarithmic scale ranging from 0.1 up to 1,000.

We have learned from, well documented nuclear accidents, 108, but theproduct is designed to address a multitude of process control units, andan estimative curve showing evolution of number of various technologicprocesses, 105, that needs to be controlled in thousands units read onright scale, 103, is given. These processes included in this curve,deliver an annual rate of accidents by several orders of magnitude overthe nuclear accidents, and number of death per year in variousindustrial major accidents, 106, readable in thousands, on the leftscale.

Number of nuclear power reactors, 107, in units, readable on rightlogarithmic scale, shows the evolution of nuclear power over time, from1945 to present Nuclear accident INES level on the left scale, 108, andwas used to calculate probability of accident 109, as a function of yearit took place, normalized at 1000 units, read on the right logarithmicscale. Because this probability came out as a zigzag line, due to poorstatistics, a trend line, 110, for the probability of nuclear accidentfunction, 108, read also on the right logarithmic scale, 103 was given.

Basically, this figure, shows the motivational background of ourinvention, simply to reduce number of accidents and to reduce theirseverity, and we intend to address the industrial and other technologicaccidents, by a better human machine interface, that to increaseoperator's and crew's awareness on the process they control, in realtime, with maximum available and necessary details in real time, inorder to reduce with at least a half the actual accident rate. Bydeveloping this method and system we involuntarily reached anothermilestone, even more profitable, that consists in the capability ofoperators to place their equipment on the maximum efficiency, and betterschedule maintenance sessions.

FIG. 2 shows a schematic diagram of an augmented reality 3D immersivevisualization control method and system meant to increase operators'awareness on the process they control being the main embodiment of ourinvention. The system contains a plurality of augmented realityheadsets, 200, that have a high resolution display, 201, with twosections, seen distinctly one by the left eye, 202, and one by the righteye, 203, using a set of lenses, 204, in order to accommodate fordistance and field of view. As shown in the state of the art section,there is an intense activity in developing these devices mainly forgaming and common user applications, and the headset itself is not anobject of our invention, because we will use some from the currentproduction which ones that will better fits our needs, and cost, but asa courtesy to those who will use our invention we described a little bithow theses stereoscopic imaging system works. An embodiment of ourinvention is the enhancement we provide to the headsets to becomecompliant with our needs for operator end data acquisition, with respectto accurate, redundant and robust localization and communication,operator bio-parameters and operator's current environment datacollection and dissemination.

No matter if the system is using an anaglyph, or shutters, polarizers,holograms, it will create a left image, 205, and a right image, 206,which operator's brain will process and have the 3D vision perception.

To the actual device, as an embodiment to our invention, we will comewith accurate inside buildings localization, and attitude device, 207,containing a 3-gyro-accelerometer-compass and inclinometer circuit,intended to produce very high localization accuracy, and in order toproduce for the computing sub-system and other users good data, it willalso contain a left imager, 208, and a right imager, 209, containingimaging cameras in visible, with filters and polarizers, illuminatingdevices, IR or UV, a set of LIDAR, radar and sonar scanners, 210,deployed based on need, used to give a clear, accurate localization, ofan object of interest.

Location sub-system on headset comprising may also have attached anultrasound distance measurement device, or a scanned ultrasound locator(proximity sonar), useful in hazardous low visibility circumstances, atunable Lidar, able to visualize various gases, a proximity radardevice, and in hazardous open areas as war zones, a telemetry detectoras radar, laser beam exposure detector might be useful, in order tobecome collective knowledge and be used by the augmented reality systemto identify the object, its status and environment, for the best realtime information menus. This system is conceived to be disaster proofsystem, and operate not only in nice rooms, but in disaster affectedinstallations. The problem is that disaster compatible mobile equipmentis more expensive and difficult to wear, than light headsets designed tooperate in laboratory or room environment, and there is no good solutionto this problem. Our opinion is that at least one of the mobile sets hasto be equipped with all hazard proof equipment based on level ofseverity anticipated by the designer or installation owners.

Augmented reality scenes, 211, ideographically presented in FIG. 2, ismeant to give reader some details on how a mobile, operator level deviceworks, where operator may have available a menu option bar for augmentedreality, 212, that activated may download from, a cloud, 214, that maybe any specialized computer or computer cluster, via local Wi-Fi, IR orcable communication, the desired data, to be put on screen. Operatorcommunicates with computer via a pointing or other operatorcommunication with compute device, 213, that can be a mouse, glove,positioning system eye tracking and EEG detector, EMG detector which bythought or muscle electric activation may perform functions asselection, mouse over or mouse click. It is not an embodiment of ourinvention, there are plenty of devices on market and we may pick andchoose which one seems more fit for the task.

Of course, an operator may have a position in front of control panel orinstrument, 215, or may be anywhere in the installation, and that haveto make no difference in the quality and accuracy of data he receivesand transmits.

In almost all the cases, we desire that operators to be out of anyinfluence, to be healthy and sound, and sober, therefore we need tocontinuously know operator bio-parameters, by using a monitoringsub-system, 216, and to know operator's environment parameters, by usinga monitoring sub-system, 217, no matter if operator is inside controlroom, 218, or somewhere in the plant, all that data have to betransmitted via data bus, 219, and be delivered to all sub-systems inreal time. The interest for knowing operator biometric parameters comesfrom the fact that in many cases the responsibility for the fate of aregion, is in the hand of very few in charge with control of a verysensitive process, and is good to know that in any moment they are up tothe job. Operator bio-parameter monitoring system comprises a pulserate-oximeter, temperature, breathing rate, to see if the vitals areworking right, in the particular range of the known operator. Skinconductivity and pH, as well electric functions as EEG, EKG, EMG, areneeded to assure that operator mental state is appropriate with theoperations, is not sleepy or have other concerns and preoccupations thatmay distract him from job. In special cases breathing content of carbondioxide, oxygen, water and alcohols vapors and body odors may bemonitored. Anomalies detected will trigger a warning to operating crewleaders and require a permission to operate and an explanation,eventually a physician report. Position of body components might be asupplementary measurement, especially when operator is in hazardousenvironment protective suits, and is associated with mobile environmentmonitoring subsystem and headset info.

Mobile operator environment monitoring system has temperature sensors,relative or absolute humidity sensor, multi-gas monitoring for area withtoxic or explosive gas leak hazards, sound and vibration monitoring ofsurround area, radiation type and level measurement for radiation hazardenvironments, or EM spectrum monitoring.

Mobile operator may supplementary have life support gear as hazmat fullprotection suit with parameters monitoring capability for air or oxygenpressure, temperature, humidity, external temperature field on hazmatsuit, local power supply control parameters for assisted heating oractuators, and/or exoskeleton control parameters.

The virtual reality enclosure 220 is built with the purpose of theproject is to have at least one person look inside all of the structureas in a doll house similar to a supervisor, using various visualizationfunctions. That to have all of the operating details necessary forexplanation for any instrument or module component of a control box andresults of complex simulations that may predict the status and futureevolution of the controlled process on mobile augmented reality devices.

Therefore all operators will be connected, and act all like one, inperfect coordination. A virtual and augmented reality plant'svisualization and control sub-system, 220, will be used in parallel withcurrent operation, by specialized, authorized operators, to gain areal-time overall view of the plant operation, 221, a detailed view ofvarious controlled processes, 222, a holistic view of plant connected togrid and environment, 223, in order to bring operation regime of allequipment to optimum. This embodiment of present invention will bringmost of the savings to pant owners, which will add to the gains from theprobability of accident and accident severity level reduction.

A high capability computing system, 224, is in charge with presentationof the entire information for operators, supervisors and approvedexternal, remote links, where customized augmented reality istransmitted to user's visualization device, generally that is preparedby a super-computer that integrates a large variety of data withsimulation

This computer structure, call it cluster, supercomputer ormicroprocessor based device sub-system will integrate data, in order togenerate simulation, and specialized visualization requests, 224, basedon models and cutting edge simulations as for example the actualprograms developed under CASL initiative, or ComSol simulations, FiniteElement or Fluid Dynamics, Robotics or CAD using the real-time datagenerated by instrumentation to initiate calculations and benchmarkthem, producing virtual images of the process and states insideequipment, unavailable by direct measurement means.

This simulation of various processes and equipment operation, 225, inplant will be used to augment current measurements acquired anddisplayed by the system.

Another computing structure, will be used as a “Memory of operatingparameters in the previous regimes”, 226, complementary to the virtualreality simulation environment previously described. This is a learningsystem that accumulates data from various previous regimes, andidentifies correlations between various parameter variations, trying topredict what might happen when one or more parameters are varying. Thissub-system runs in parallel with simulation sub-system because it issimulating system using previous operating regime data, 227, and may runwithout interfering with equipment control system, its results have tobe considered by operators to use in their actual control. Theinstallation may also be used to benchmark various simulation codes,because these codes may have available same inputs and their predictionshave to be compatible with measurements, or the differences will bevaluable indicators on what is missing or malfunctioning inside a code.

During various event, operators need to know, where are located variousinstruments or pieces of equipment, as for example during Fukushimanuclear reactor power plant accident, it took more than 4 hours untilthe crew to locate and try to operate a pressure safety valve. In thiscase, the building and equipment plans will be put together in a 3Dversion of the digital plant, creating the so called instrumentpositions log, in plant's coordinates; 228. Some simulations ofdestruction, as enemy bombardment, earthquake, flooding may be performedin order to anticipate abnormal instrument positions, and potentialassociated hazards.

But, having building and installation plans, 229, only is not enough ifthese are not correlated with object identification and coordinatelocation system. This subsystem uses this information available to useguidance to operators and maintenance teams to locate and findinstruments and pieces of equipment, get all data about them, from howthat objects are used in the process to electronic and mechanicaldiagram, user manuals, repairing manuals, revisions and certifications,and other data form similar instruments. This subsystem may evencoordinate and control a guided tour of visitors, detect unauthorizedpresence in various locations based on image recognition or RFID (RadioFrequency Identification Device) reading, or occurrence of unusual noiseor images, related to presence of animals, intruders, displacements,overheating, etc.

In many accidents, the lack of a robust communication system, sealed thefate of that controlled process, therefore we are introducing anintegrated, redundant communication system, 230, where mobile operatorsand those assigned to fix positions to be able to share real timecomplex information, and act all like a super-smart one.

Image, sound, vibration monitoring sub-system, 231, is in fact adistributed assembly of various receivers, placed in fixed locations,with acknowledged importance, mobile systems as operators' headsets, orrobotic devices on surface and in air as drones, that work together andmake a data fusion, providing an integrated image of the system, withtriggers that make an instance or flag to pop up into assigned operatorattention.

Data acquisition and processing sub-system, 232, is the core of thesystem, because it accumulates in real-time the data from all theinstruments, and in parallel with the main display it delivers it tooperators, and to simulation sub-systems. Data fusion, and data drivenplant operation and control is important, and relies on this computersub-system. All data sources, measurement instruments and othertransducers, as cameras, and communication systems, have to be providedwith redundant power and local backup, in order to preventinstrumentation blackout, similar to what happened during Fukushima,Daichi 4, tsunami flooding.

FIG. 3 shows how a method to increase industrial-process operatorsawareness on a controlled process and control equipment andinfrastructure using augmented and virtual reality works using a virtualreality visualization enclosure, 300, where the images are produced by aComputer, 301, triggered by operator, 302, request for analysis, at aterminal, 303, or by emergencies, 306, generated by the processmonitoring system, 304, which can interrupt or overlap in the currentvisualization session, 305.

A set of augmented reality mobile devices, 310, used in coordinationwith an internal location system, 311, that renders mobile operatorsaugmented images based on operators' location, 312, and line of sight,313.

Operator request for information or process simulation transmitted viacommunication with computer devices, 314.

A computer system, 304, that continuously acquires data from the processand compares with good operation margins obtained after storing andprocessing previous operation history, 315, and presents it to operator.

A process and instrument database, 316, that stores all the informationrelated to equipment and infrastructure and processes, stores andrenders it based on various criteria selected by operator, 317, ortriggered by events computer 311.

A simulation computing system, 317, that uses theoretical formulas andinstantaneous instrument data, 318, to guess the most probable state ofvarious parameter distribution inside the equipment and check for goodstanding and optimum operation regimes.

A redundant communication system, 314, 319, design to assureinstantaneous communication and data exchange among all the teammembers, 302, 320-322, based on a predefined need to know, or on-spotdemands for information, as part of a personal communication setassigned to a person with multiple location calibration points, multipleconnection capabilities, and event alerts in the system.

It describes an operator biometric parameters and surroundingenvironmental parameters measuring sub-system that also monitors hislife support and hazardous conditions, 323, that takes the measured datain all the above mentioned sub-systems, 316, and, stores it in a database, 315, of memory of operation parameters.

The line of sight, 313, or a specific request, 317 are used to triggerthe start points of various simulations from equipment 318 to assembly,transfer the data to various specialized computers, 316, 311, 304, 303,to create virtual reality presentation 305, to supervise and optimizethe operation regime of the controlled process. Transfer the data tovarious computer systems, 316, 311, 304, 303, is used to generateaugmented reality to mobile devices, 310, worn by operators 320, 321,322, that, localize the operator position, 312, and headset line ofsight, 313, and object of interest, 318, where it makes imagerecognition of the object of interest, 318, or at operator's request,317, based a menu list generates:

-   -   Descriptions of the object of interest;    -   Details of its status of operation;    -   Differences of parameters between measured and predicted;    -   Other information on the object present in data base;    -   Synchronizes with other team members;    -   Acquires data on operator's surroundings and transmits to        database for general use;    -   Monitors operator's parameters;    -   Transmits operator various complex simulation scenarios and        possible decision option outcomes;    -   Assures various communications, 314, 311, among team members        302, 320, 321, 322;    -   Informs team members on the optimum operations;    -   Alerts team members on potential events and emergencies appeared        in the system, 306.

This assembly of hardware and software creates operator awarenessincrease in a system comprising a plurality of augmented realityvisualization devices 303, 310 made of a virtual reality visualizationenclosure 300, comprising, one or several individual visualizationdevice placed on individuals 310, or placed on an enclosure, 303, wheretheir movements and requests have to be correlated with the simulatedimage on each individual display.

A computing system 315, 317, 301, 304, 311, 316, is creating the virtualreality on demand that is provided to any operator in enclosure, 300,connected via equipment to computing system and/or operatingindependently, 320, 321, 322.

FIG. 4 describes an augmented reality mobile device 411, being, 310 inFIG. 3, on operators on move inside the controlled infrastructure or atcontrol panel made of a head-on visualization device using camera, 422,placed on operator, 400, and projected on a device transferring it inoperator's eyes, overlapped by virtual reality images generated by aremote computer.

Because operator location and its line of sight location accuracy isimportant, each local room has an entry location system, 402, that workssynchronized with the gyroscope, 414, and accelerometers, 416,inclinometers, 415, magnetometers, 417, on the operators head, 410, ableto recalibrate in each individual room equipped with a fixed reference,403, in access area, 402, where the operator, 400, looks and presses,401, calibration mode 407, button, making a location system at fixedreference, 408, that localized, 409, the operator's, 400, headset, 411,and its direction, 410, by identification of headset, 411, based onRFID, 412, and communication with headset, 404, 405, 406, exchangingcoordinates.

On the head-set 411, there is an inertial base, with GPS, 413, onheadset, and also comprising a 3 axis gyroscope, 414, a 3 directionsaccelerometers, 416, a 3 direction inclinometers, a 3 directionmagnetometer, 417.

A location system on headset, 411, is also comprising a stereoscopiccamera system 422, in various bands as visible, filtered, IR-nightvision or IR-thermo-vision, an ultrasound distance measurement, 419, oran ultrasound locator (proximity sonar), 419, an EM spectrumanalyzer/receiver, 418, a Lidar, 421, a Radar, 420, acting also as atelemetry detector as radar, laser detector.

A redundant communication system based on room/area level WiFi, 405, andIR emitter-receivers, 406, where the headsets are also equipped withcompatible WiFI and IR receiver emitters, and cable connection ports,404, in the area assuring communication management system in headset andcentral location with robust communication capabilities.

FIG. 5 describes a mobile operator, 500, bio-parameter monitoring systemthat is distributed on operator's body, 500, comprising a pulserate/EKG, 514, a temperature measurement devices, 512, distributed onthe body, 500, or in a most preferred singular location, an Oxygen levelmeasurement, 510, correlated with breathing gas support 511, andmeasuring Breathing rate, 513, a skin conductivity, 509, and pHmeasurement, 517, correlated with electric functions (EEG, EKG, EMG),516, and a distributed on body, 500, position of body componentsmeasurement 515.

Operator, 500, wearing an augmented reality headset, 507, is sitting ona fixed reference area, 501, in front of the entry wall, 502, with arm,503, pressing the button, 504, in order that the room reference systemof coordinates 505, to establish a communication 506, and recalibrateand transmit all parameters of interest.

FIG. 6 gives details on mobile operator, 600, environment monitoringsystem that is comprising, temperature sensors, 604, humidity sensor,606, multi-gas monitoring sensor, 605, sound and vibration surroundmonitoring sensors 607, 611, radiation type and level measurement 608,EM spectrum monitoring, 609.

Mobile operator, 600, supplementary life support gear parametersmonitoring for air or oxygen pressure, temperature, humidity, 612,correlated with external temperature field, 604, on hazmat suit, 614,including power supply control parameters for assisted heating oractuators, and Exo-skeleton control parameters, 610.

Operator headset, 603, pointing system and communication interface atoperator choice, as joystick, eye tracker, 602, as part of augmentedreality gear, including voice recognition.

FIG. 7 describes an operational situation, with 3 rooms, 700, 701, and709 where the wireless communication from rooms 700 and 701 is notaccessible in the room 709 due to shielding in the wall 708. Each localroom has an entry location system, 703, that is used by the operator,705, as soon as he penetrated the room, 700, through the door, 706, hesteps in front of the location panel, 703, touches the calibrationbutton with the hand 704, while having the direct communication cable,707, connected. In this moment the coordinate system in his helmet andthe room's coordinate system, 702, are synchronized, and he got thecoordinates r, valid inside rooms 700, and 701. The same operator oranother one, as soon as he passes through the wall 708, that negativelyaffects communication from near by room, 701, have to report himself atcoordinate calibration entry point 710, that is connected via the wire711 to the network, that includes the port 703, also. The calibrationprocedure is identical; operator, 715, connects the cable, 716, pressesthe button with the hand, 714, and is assigned to the room, 709,reference system, 712, with a new position vector r′, 713, and alsoassigned to communication means active in that room. This is basicallythe procedure that helps keep track and have real-time, robust,communication with all operators no mater where they are located in thefield.

FIG. 8 shows a computer system, 800, controlling the information flowwith each headset, 801, and operator system, 802, and plant's computers,803, 804, 805.

Plant level distributed computer structure, 803, 804, and data base,805, which is processing data upon operator's request and istransmitting it to operator's headset, 801, as augmented and virtualreality content via its assigned communication computer, 806, that ispreparing virtual and augmented reality display, 801.A computer cluster, 808, that is concentrating data from varioussimulations and data acquisition sub-system and is preparing to beavailable to be sent to operator's augmented and virtual realitycomputer;

A simulation unit, 802, 803, that is using theoretical models on processand associated equipment in order to simulate its operation, and use theprocess measured data to benchmark and as initiating points forcalculations;

A simulation and prediction data processing unit, 803, 804, 805, thatuses a data base, 808, formed during previous operating regimes of thatplant or similar ones, to predict the domains of likelihood of eachparameter for a certain configuration and to signal anomalies;A data acquisition sub-system, 807, that collects memorize and organizein databases the process generated data from instruments and controls,and prepares it for further use;A subsystem, 803, that is acquiring and processing, image, sound andvibration data and prepares to be sent to operators via communicationsystem, 806;A sub-system that assures communication, 806, among operators, 801, 802,and computing system, in various regimes at demand;A computing subsystem, 808, that stores in data bases the informationabout the plant layouts, instruments and equipment positioning anddetails, and generates guidance and all information to operators, 801,and maintenance teams.

EXAMPLES OF THE INVENTION

The current idea is a 3D digital simulation of the plant will be able tohelp plant's operators safely operates the equipment in any conditions.

The present patent is not intended to drastically change how the plantis operated, leaving in place the same control room, with the samecontrols, but to add a new layer of control based on data fusion andsimulation, driving to a data driven operation.

When the present invention is applied to a power plant, based on coal,oil, etc., we think that the processes have to be sorted based on theneed to react to them, or on emergency. For example the variation of thepower demand and the fuel to feed the boiler may be more urgent, thanthe operations of purchasing fuel for reserves. Also in case of failureof a pump, an operator have to go to that point to learn more about thecircumstances while the main room have to immediately deal with thatreducing the power or starting auxiliary pumps . . . . In thiscircumstances a simulation of the parameter's evolution, pre-calculatedand updated for the current data will be a good guidance for operatorsto decide if only a supplementary pump have to be started or anemergency shutdown is required in order to prevent further damage. Ifthere are several power groups, working together, this information aboutthe failure have to be transmitted and they have to increase their powerlevels to meet the demand, or transmit the information up in the networkthat a dispatcher to make real time decision.

Various simulations will show operator what should be happening in thefuture if a decision or another is taken, compared to what is actuallyhappening and will be able to notify the operator if something is wrong.The simulation will also tell the operator what the best way to getmaximum power, and how to take corrective action if something does gowrong.

The 3D augmented reality immersive system offers using field data fusionwith predictive simulations a holistic view of the nuclear reactorstate, safety and system health and advices possible future paths.

This is useful for nuclear reactor operators to detect in real time anyissue in the reactor, to optimize maintenance and have minimized thehuman factor instrumentation probability.

The system can also later be built to where it runs the whole nuclearpower plant autonomously creating less risk of human error.

For example in nuclear power production by buying this product thecustomer will reduce the human factor contribution to a nuclear eventmaking it end up in an incident, rather than becoming an accident.

The probability of nuclear accident is now of about 0.07%, that means anaverage cost of $5B per year and an average cost of insurance perreactor of $50 M/y per power plant.

Supposing that this system will reduce the probability of accident orits magnitude by only 10% it will give a gain of about $5M/year perplant, and possible cover in one year or two, the cost of this newequipment. In nuclear power plant, the complexity is higher, becausenuclear reactor is more complex and difficult to operate than a usualheat source used in conventional power plants, and the role of suchequipment might become even more important.

Other plants that may successfully use this augmented and virtualreality system, are in oil processing, chemistry, foundries,manufactures, etc.

The invention claimed is:
 1. A system to increase operator awarenesscomprising: a. Plurality of augmented reality visualization devices madeof: I. Virtual reality visualization enclosure comprising: i. One orseveral individual visualization device placed on individuals inside anenclosure, where their movements and requests are correlated with asimulated image on each individual display; ii. A computing systemcreating a virtual reality on demand provided to any operator inenclosure connected to the equipment computing system and operatingindependently; II. Augmented reality mobile devices located on operatorsmoving inside the controlled infrastructure or at control panel, madeof: i. A head-on visualization device using camera placed on operatorand projected on a device transferring it in operator's eyes, overlappedby virtual reality images generated by a computer; ii. A local roomlocation system, that works synchronized with a gyroscope andaccelerometers, inclinometers, magnetometers on the operators head, ableto recalibrate in each individual room comprising: a. Fixed reference inaccess area where the operator looks and presses calibration mode; b.Location system at fixed reference that localized the operator'sheadset, and its direction; c. Identification of headset, based on RFIDand communication with headset exchanging coordinates; d. Inertial baseon headset, comprising: i. 3 axis gyroscope; ii. 3 directionsaccelerometers; iii. 3 direction inclinometers; iv. 3 directionmagnetometer; e. Location system on headset comprising: i. Stereoscopiccamera system in various bands as visible, filtered, IR-night vision orIR-thermovision; ii. Ultrasound distance measurement; iii. Ultrasoundlocator (proximity sonar); iv. Lidar; v. Radar; vi. Telemetry detectoras radar, laser detector; f. Redundant Communication system comprising:i. Room/area level WiFi and IR emitter-receivers; ii. Headset WiFi andIR receiver emitters; iii. Cable connection ports in the area; iv.Communication management system in headset and central location; g.Mobile operator bio-parameter monitoring system comprising: i. Pulserate; ii. Temperature; iii. Oxygen level; iv. Breathing rate; v. Skinconductivity and pH; vi. Electric functions (EEG, EKG, EMG); vii.Position of body components; h. Mobile operator environment monitoringsystem comprising: i. Temperature sensors; ii. Humidity sensor; iii.Multi-Gas monitoring; iv. Sound and vibration surround monitoring; v.Radiation type and level measurement; vi. EM spectrum monitoring; i.Mobile operator supplementary life support gear parameters monitoringcomprising: i. Air or oxygen pressure, temperature, humidity; ii.External temperature field on hazmat suit; iii. Power supply controlparameters for assisted heating or actuators; iv. Exo-skeleton controlparameters; j. Operator headset pointing system and communicationinterface at operator choice, as; i. joystick, ii. eye tracker, iii.voice recognition; III. A computer system controlling information flowwith each headset and operator system and plant's computers; b. Plantlevel distributed computer structure and data base, which is processingdata upon operator's request and is transmitting it to operator'sheadset as augmented and virtual reality content via its assignedcommunication computer that is preparing virtual and augmented realitydisplay; c. A computer cluster that is concentrating data from varioussimulations and data acquisition sub-system and is preparing to beavailable to be sent to operator's augmented and virtual realitycomputer; d. A simulation unit that is using theoretical models onprocess and associated equipment in order to simulate its operation, anduse process measured data to benchmark and as initiating points forcalculations; e. A simulation and prediction data processing unit, thatuses a data base formed during previous operating regimes of that plantor similar ones, to predict domains of likelihood of each parameter fora certain configuration and to signal anomalies; f. A data acquisitionsub-system that collects memorize and organize in databases the processgenerated data from instruments and controls, and prepares it forfurther use; g. A subsystem that is acquiring and processing, image,sound and vibration data and prepares to be sent to operators viacommunication system; h. A sub-system that assures communication amongoperators and computing system, in various regimes at demand; i. Acomputing subsystem that stores in data bases the information about theplant layouts, instruments and equipment positioning and details, andgenerates guidance and all information to operators, and maintenanceteams.
 2. A system to increase operator awareness according claim 1,that uses a redundant local network to connect the operator headsets andassociated monitoring devices that provide case sensitive or on demandaugmented and virtual reality to operators.
 3. A system to increaseoperator awareness according claim 1, where the communication system iscontrolled by computer acting in closed circuit, in various regimes. 4.A system to increase operator awareness according claim 1 where operatorcoordinate location system has calibration points in several locationsaround the plant.
 5. A system to increase operator awareness accordingclaim 1 having specialized gear to monitor operator's biometricparameters in a modular fashion depending on plant specificrequirements.
 6. A system to increase operator awareness according claim1 where the simulating computing system that uses theoretical modelsruns independently of the simulating system that uses previous operatingparameters of that or similar process to predict future evolution.
 7. Asystem to increase operator awareness according claim 1 where thecontrol room and process control procedures are not changed by theapplication of the operator augmented and virtual reality system.
 8. Asystem to increase operator awareness according claim 1 that integratesall acquired data from instruments, video, audio and vibration to createthe augmented reality.
 9. A system to increase operator awarenessaccording claim 1 that uses various modes to achieve the communicationbetween operator and computing system.
 10. A system to increase operatorawareness according claim 1 that does not interfere with the controls ofthe process, and is the operator the person that actuates the controlsin the same manner as without the system.
 11. A system to increaseoperator awareness according claim 1 where the operator's headset havebeen equipped with local installation location devices and stereoscopicimaging.